Repository navigation
Conversation
|
Follow-up validation at
Eight H200 GPUs in one NVLink domain; deterministic compact BF16; exact CPU receive counts; D=3072, F=1280, top-8, 16 local experts and 16 communication SMs. All three arms use the same candidate build. Three fresh processes, each with 12 blocks covering all six arm orders twice, 20 calls/arm/block and 20 warmup calls/arm. The metric is synchronized wall time through device completion, including CPU orchestration, dispatch, consumer and release: median across 36 maximum-rank block means per arm/shape.
All three fresh-process median comparisons improve at 8192/16384 and regress at 1024, including versus deferred. This remains opt-in; the small-token regression is not hidden. These are one-node component timings, not full-training or overlapped-batch throughput. Fixed routing-plan construction, final expert projection, combine, backward and optimizer are outside the measurement. The exclusive same-buffer lease and forward-only lifetime restrictions are unchanged. Validation passed 256 distributed lease rank/cases (16 combinations × 8 ranks × both Provenance: Torch 2.13.0+cu132, CUDA 13.2, actually loaded NCCL 2.32.3. This reuses the existing validated native extension from New report, exact protocol, CPU-only audit and GPU reproduction instructions. The gist contains the new benchmark/analyzer, raw per-rank logs, frozen manifest and analysis; all historical evidence is retained. This does not revise the distinct combined DeepGEMM measurements or the separate prefix-sort prototype results. |
9e8f582 to
71157be
Compare
1 participant
dispatch(..., borrow_recv=True)skips the activation copy of the dispatch epilogue: the epilogue writes only metadata plus a row map and returns aDispatchRecvView(slab,row_indices) over the communication buffer, which an indexed consumer such as a gathered GEMM reads directly. The view holds an exclusive lease:release(*streams)orders the next communication after the consumers, and dispatch/combine/LB calls/destroy()reject an active view. Supported: fresh compact BF16 dispatch within one NVLink domain with CPU counts anddefer_epilogue=False; everything else is rejected up front. Default behavior is unchanged.8×H200, deterministic, hidden 3072, top-8, 16 local experts, dispatch plus the same indexed consumer: −13.1% at 8192 tokens/rank and −13.5% at 16384, but +15.9% at 1024, hence opt-in. Fixed-route serial forward only; no overlap or training-throughput claim. Combined with DeepGEMM #463 in a live EP8 + gate/up GEMM + SwiGLU step: −17%. Raw cycles and the reproducer: gist.
Test:
python tests/ep/test_recv_view.py --num-processes 8(bitwise payload/metadata against the gathered inputs, replay/combine after release, delayed cross-stream readers, invalid-mode rejection).Re-validated 2026-10-16 at 71157be (native code byte-identical to the measured build) on 8×H200, PyTorch 2.13.0+cu132:
test_recv_view.py16/16 cases under bothEP_AVOID_RECORD_STREAMpolicies; the EP8 functional suite passes with CPU counts for direct and deferred epilogues. With--do-cpu-sync 0the unmodified suite stops at its full-capacityrecv_x_twicecomparison on this branch exactly as on upstream main; with the valid-prefix comparison from #786 it passes on both.